reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerSimon McVittie <smcv@debian.org>
Sun, 10 Sep 2023 14:51:04 +0000 (15:51 +0100)
commit16f43d471f7afee1ed975590470bff802f989707
tree9459e41358f17edc27c43665577ff12b820be227
parent96a8f1df2a2aebec7ec67341a0577ad732ec7940
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, with additional inspiration
from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/reftests/gtk-reftest.c